Staged Insert (direct write mode):
- stage_object() context manager for writing directly to storage
- StagedObject provides fs, store, full_path for Zarr/xarray
- Cleanup on failure, metadata computed on success
- Avoids copy overhead for large arrays

ObjectRef fsspec accessors:
- fs property: returns fsspec filesystem
- store property: returns FSMap for Zarr/xarray
- full_path property: returns full URI

Updated immutability contract:
- Objects immutable "after finalization"
- Two insert modes: copy (existing data) and staged (direct write)

- Use dedicated staged_insert1 method instead of co-opting insert1
- Add StagedInsert class with rec dict, store(), and open() methods
- Document rationale for separate method (explicit, backward compatible, type safe)
- Add examples for Zarr and multiple object fields
- Note that staged inserts are limited to insert1 (no multi-row)

- Filename is always {field}_{token}{ext}, no user control over base name
- Extension extracted from source file (copy) or optionally provided (staged)
- Replace `original_name` with `ext` in JSON schema and ObjectRef
- Update path templates, examples, and StagedInsert interface
- Add "Filename Convention" section explaining the design

- Rename store metadata: dj-store-meta.json → datajoint_store.json
- Move objects/ directory after table name in path hierarchy
- Path is now: {schema}/{Table}/objects/{pk_attrs}/{field}_{token}{ext}
- Allows table folders to contain both tabular data and objects
- Update all path examples and JSON samples

- Hash is null by default to avoid performance overhead for large objects
- Optional hash parameter on insert: hash="sha256", "md5", or "xxhash"
- Staged inserts never compute hashes (no local copy to hash from)
- Folders get a manifest file (.manifest.json) with file list and sizes
- Manifest enables integrity verification without content hashing
- Add ObjectRef.verify() method for integrity checking

- Enables bidirectional mapping between object stores and databases
- Fields are informational only, not enforced at runtime
- Alternative: admins ensure unique project_name across namespace
- Managed platforms may handle this mapping externally

- Legacy attach@store and filepath@store use hidden ~external_* tables
- New object type stores all metadata inline in JSON column
- Benefits: simpler schema, self-contained records, easier debugging
- No reference counting complexity

- Add fsspec>=2023.1.0 as core dependency
- Add optional dependencies for cloud backends (s3fs, gcsfs, adlfs)
- Create new storage.py module with StorageBackend class
  - Unified interface for file, S3, GCS, and Azure storage
  - Methods: put_file, get_file, put_buffer, get_buffer, exists, remove
- Refactor ExternalTable to use StorageBackend instead of protocol-specific code
  - Replace _upload_file, _download_file, etc. with storage backend calls
  - Add storage property, deprecate s3 property
- Update settings.py to support GCS and Azure protocols
- Add deprecation warning to s3.py Folder class
  - Module kept for backward compatibility
  - Will be removed in future version

This lays the foundation for the new object type which will also use fsspec.

This commit adds a new `object` column type that provides managed file/folder
storage with fsspec backend integration. Key features:

- Object type declaration in declare.py (stores as JSON in MySQL)
- ObjectRef class for fetch behavior with fsspec accessors (.fs, .store, .full_path)
- Insert processing for file paths, folder paths, and (ext, stream) tuples
- staged_insert1 context manager for direct writes (Zarr/xarray compatibility)
- Path generation with partition pattern support
- Store metadata file (datajoint_store.json) verification/creation
- Folder manifest files for integrity verification

The object type stores metadata inline (no hidden tables), supports multiple
storage backends via fsspec (file, S3, GCS, Azure), and provides ObjectRef
handles on fetch with direct storage access.

Remove unused mimetypes imports from objectref.py and storage.py,
remove unused Path import and generate_token from staged_insert.py,
and fix f-string without placeholders in objectref.py.

- Create comprehensive object.md page covering configuration, insert,
  fetch, staged inserts, and integration with Zarr/xarray
- Update attributes.md to list object as a special DataJoint datatype
- Add object_storage configuration section to settings.md
- Add ObjectRef and array library integration section to fetch.md
- Add object attributes and staged_insert1 section to insert.md

Apply ruff formatter changes for consistent code style.

- schema_object.py: Test table definitions for object type
- test_object.py: Comprehensive tests covering:
  - Storage path generation utilities
  - Insert with file, folder, and stream
  - Fetch returning ObjectRef
  - ObjectRef methods (read, open, download, listdir, walk, verify)
  - Staged insert operations
  - Error cases
- conftest.py: Object storage fixtures for testing

Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>

Add bool and boolean as type aliases mapping to MySQL tinyint.
Update tests and documentation accordingly.

- Make size field optional (nullable) for large hierarchical data
- Add Performance Considerations section documenting expensive operations
- Add Extension Field section clarifying ext is a tooling hint
- Add Storage Access Architecture section noting fsspec pluggability
- Add comprehensive Zarr and Large Hierarchical Data section
- Update ObjectRef dataclass to support optional size
- Add test for Zarr-style JSON with null size

Clarifies the architectural distinction between the object type (AUS)
and filepath@store (external references) to address reviewer question
about multi-cloud scenarios.

Remove lazy initialization pattern for storage attribute since it was
being initialized in __init__ anyway. Storage is now a regular instance
attribute instead of a property.

- HDF5 requires random-access seek/write operations incompatible with
  object storage's PUT/GET model
- Staged inserts work with chunk-based formats (Zarr, TileDB) where
  each chunk is a separate object
- Added compatibility table and HDF5 copy-insert example
- Recommend Zarr over HDF5 for cloud-native workflows

…alidation

All 24 object storage test failures were due to test fixtures not creating
the directories they configured. StorageBackend validates that file protocol
locations exist, so fixtures must create them.

- conftest.py: Create test_project subdirectory in object_storage_config
- test_update1.py: Create djtest subdirectories in mock_stores_update

Test results: 520 passed, 7 skipped, 0 failures ✓

Add blank line after import statement per PEP 8 style guidelines.

- Removed hardcoded 'objects' directory level from build_object_path()
- Updated path pattern comment to reflect new structure
- Updated all test expectations to match new path format

Previous path: {schema}/{table}/objects/{key}/{file}
New path: {schema}/{table}/{key}/{file}

The 'objects' literal was a legacy remnant intended for future tabular
storage alongside objects. Removing it simplifies the path structure
and aligns with documented behavior.

Verified:
- All test_object.py tests pass (43 tests)
- All test_npy_codec.py tests pass (22 tests)
- All test_hash_storage.py tests pass (14 tests)

- Updated SchemaCodec._build_path() to accept store_name parameter
- _build_path() now retrieves partition_pattern and token_length from store spec
- ObjectCodec and NpyCodec encode methods pass store_name to _build_path
- Enables partitioning configuration like partition_pattern: '{mouse_id}/{session_date}'

This allows organizing storage by experimental structure:
- Without: {schema}/{table}/{mouse_id=X}/{session_date=Y}/...
- With: {mouse_id=X}/{session_date=Y}/{schema}/{table}/...

Partitioning makes storage browsable by subject/session and enables
selective sync/backup of individual subjects or sessions.

The partition_pattern was not preserving the order of attributes specified
in the pattern because it was iterating over a set (unordered). This caused
paths like 'neuron_id=0/mouse_id=5/session_date=2017-01-05/...' instead of
the expected 'mouse_id=5/session_date=2017-01-05/neuron_id=0/...'.

Changes:
- Extract partition attributes as a list to preserve order
- Keep a set for efficient lookup when filtering remaining PK attributes
- Iterate over the ordered list when building partition path components

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Added helper functions for safe 0.14.6 → 2.0 migration using parallel schemas:

New functions in datajoint.migrate:
- create_parallel_schema() - Create _v20 schema copy for testing
- copy_table_data() - Copy data from production to test schema
- compare_query_results() - Validate results match between schemas
- backup_schema() - Create full schema backup before cutover
- restore_schema() - Restore from backup if needed
- verify_schema_v20() - Check if schema is 2.0 compatible

These functions support the parallel schema migration approach which:
- Keeps production untouched during testing
- Allows unlimited practice runs
- Enables side-by-side validation
- Provides easy rollback (just drop _v20 schemas)

See: datajoint-docs/src/how-to/migrate-to-v20.md

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Added helper function for migrating external storage pointers when copying
production data to _v2 schemas during git branch-based migration.

Function: migrate_external_pointers_v2()
- Converts BINARY(16) UUID → JSON metadata
- Points to existing files (no file copying required)
- Enables access to external data in _v2 test schemas
- Supports deferred external storage migration approach

Use case:
When using git branch workflow (main: 0.14.6, migrate-to-v2: 2.0), this
function allows copied production data to access external storage without
moving the actual blob files until production cutover.

Example:
  migrate_external_pointers_v2(
      schema='my_pipeline_v2',
      table='recording',
      attribute='signal',
      source_store='external-raw',
      dest_store='raw',
      copy_files=False  # Keep files in place
  )

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

- Remove trailing whitespace from SQL query
- Remove unused dest_spec variable
- Fix blank line whitespace (auto-fixed by ruff)

Auto-formatted by ruff-format to collapse multi-line function calls

Unified stores configuration with configurable prefixes and filepath_default

Replace deprecated 'external storage' terminology with canonical terms:
- 'object storage' for general concept
- 'in-store storage' for @ modifier specifics
- 'in-table storage' for database storage

Changes:
- builtin_codecs.py: Update BlobCodec, AttachCodec, HashCodec docstrings
  * 'internal/external' → 'in-table/in-store'
  * Update examples and get_dtype() docstrings
- settings.py: Update StoresSettings docstrings
- gc.py: Update module docstring and format_stats()
- expression.py: Update to_dicts() docstring
- heading.py, codecs.py, declare.py: Update internal comments
- migrate.py: Add note explaining use of legacy terminology

Ref: TERMINOLOGY.md, DOCSTRING_TERMINOLOGY_REPORT.md

Replace deprecated SQL-derived terms with accurate DataJoint terminology:
- 'semijoin/antijoin' → 'restriction/anti-restriction'
- Clarify that A & B restricts A (does not join attributes)

Changes in source code comments:
- expression.py:1081: 'antijoin' → 'anti-restriction'
- condition.py:296: '(semijoin/antijoin)' → 'for restriction'
- condition.py:401: '(aka semijoin and antijoin)' → removed

Rationale: In relational algebra, joins combine attributes from both operands.
DataJoint's A & B restricts A to matching entities—no attributes from B appear
in the result. This is fundamentally restriction, not a join operation.

Clarify dual-mode codecs in builtin_codecs docstring

Replace all uses of uint8, uint16, uint32, uint64 with their signed
counterparts in test schemas, following the removal of unsigned types
from core types.

Changes:
- uint8 → int16 (larger signed type for unsigned 8-bit range)
- uint16 → int32 (larger signed type for unsigned 16-bit range)
- uint32 → int64 (larger signed type for unsigned 32-bit range)
- int unsigned → int64 (native type replacement)

Updated files:
- tests/schema_simple.py: E.M.id_m
- tests/schema_type_aliases.py: Remove unsigned types from test table
- tests/schema.py: Auto.id, Ephys.Channel.channel
- tests/schema_university.py: Student.student_id, Course.course
- tests/integration/test_type_aliases.py: Remove unsigned type tests
- tests/integration/test_hidden_job_metadata.py: All uint8 → int16

Note: test_blob.py and test_blob_matlab.py unchanged (testing numpy
dtypes in serialization, not DataJoint table definitions).

Part of v2.0 core type system revision.

- priority: uint8 → int8
- pid: uint32 → int32
- connection_id: uint64 → int64

Ensures job queue uses only core signed integer types.

- Tests that 'int unsigned', 'bigint unsigned', etc. are allowed
- Documents that these are MySQL-specific, not portable
- Users should prefer signed core types for compatibility

Also bump version to 2.0.0a22

…types

Remove unsigned integer types from core types

- Replace NOW() with CURRENT_TIMESTAMP in all SQL queries
- NOW() is MySQL-specific, CURRENT_TIMESTAMP is SQL standard
- Update comments referencing NOW() to CURRENT_TIMESTAMP
- Both MySQL and PostgreSQL support CURRENT_TIMESTAMP
- Tests pass with MySQL (PostgreSQL compatibility verified)

Files changed:
- src/datajoint/jobs.py: 7 occurrences replaced
- src/datajoint/autopopulate.py: 2 occurrences replaced
- tests/integration/test_autopopulate.py: 1 comment updated

…stamp

Replace NOW() with CURRENT_TIMESTAMP for PostgreSQL compatibility

)

* chore: remove obsolete datajoint.pub from old plugin system

* chore: remove integration_test_summary.txt and add to gitignore

* fix: Replace NOW() with CURRENT_TIMESTAMP for PostgreSQL compatibility

- Replace NOW() with CURRENT_TIMESTAMP in all SQL queries
- NOW() is MySQL-specific, CURRENT_TIMESTAMP is SQL standard
- Update comments referencing NOW() to CURRENT_TIMESTAMP
- Both MySQL and PostgreSQL support CURRENT_TIMESTAMP
- Tests pass with MySQL (PostgreSQL compatibility verified)

Files changed:
- src/datajoint/jobs.py: 7 occurrences replaced
- src/datajoint/autopopulate.py: 2 occurrences replaced
- tests/integration/test_autopopulate.py: 1 comment updated